Protecting Xen hypercalls Intrusion Detection/ Prevention in a Virtualization Environment

During the past few years virtualization has strongly reemerged from the shadow of the mainframe generation as a promising technology for the new generation of computers. Both the research and industry communities have recently looked at virtualization as a solution for security and reliability. With the increased usage and dependence on this technology, security issues of virtualization are becoming more and more relevant. This thesis looks at the challenge of securing Xen, a popular open source virtualization technology. We analyze security properties of the Xen architecture, propose and implement different security schemes including authenticated hypercalls, hypercall access table and hypercall stack trace verification to secure Xen hypercalls (which are analogous to system calls in the OS world). The security analysis shows that hypercall attacks could be a real threat to the Xen virtualization architecture (i.e., hypercalls could be exploited to inject malicious code into the virtual machine monitor (VMM) by a compromised guest OS), and effective hypercall protection measures can prevent this threat. The initial performance analysis shows that our security measures are efficient in terms of execution time and space.